Smallest eigenvalue distributions for two classes of /3-Jacobi ensembles 

Ioana Dumitriu* 
August 16, 2011 



Abstract 

We compute the exact and limiting smallest eigenvalue distributions for a class of /?- Jacobi ensembles 
not covered by previous studies. In the general (3 case, these distributions are given by multivariate 

2 IB 

hypergeometric 2-^1 functions, whose behavior can be analyzed asymptotically for special values of p 
which include f3 e 2N+ as well as for j3 — 1. Interest in these objects stems from their connections (in the 
f3 = 1,2 cases) to principal submatrices of Haar-distributcd (orthogonal, unitary) matrices appearing 
in randomized, communication-optimal, fast, and stable algorithms for eigenvalue computations [8], [4|. 

1 Introduction 

This paper was born from the author's interest in the following problem. 

Motivating Problem (MP). Given annxn Haar-distributed orthogonal or unitary matrix U n , 
choose a principal submatrix M r formed by r rows and r columns, r < n/2. Since the Haar (uniform) 
distribution is invariant under permutations of rows and columns, we might as well assume that M r is 
the upper-left corner principal submatrix (in Matlab notation, M r = U n (l : r, 1 : r)). 

1. What is the distribution of the smallest singular value of M r l 

2. If r, n — > 00, how does the singular value asymptotically depend on n and r? 

3. In particular, what kind of asymptotics do we get when r = 0(n)? 

The MP arose in the author's work in scientific computing, namely on (parallelizable, fast, and 
stable) randomized algorithms 8 , 4 , since the quality of the randomized rank-revealing decomposition 
RURV presented there depends on the order of the said smallest singular value distribution. The main 
interest in that work is for the case when n and r are extremely large (the matrix U n itself does not 
fit into the processor's cache memory); hence Question 2. The reason why Question 3 is of particular 
importance is because, in that context, r represents the split following a "Divide-and-Conquer" step in 
an eigenvalue algorithm (and we must expect that r will be a fraction of the total number of eigenvalues 
n). 

The distribution, naturally, depends on whether the matrix is orthogonal or unitary. Question 1 
relates to work by Collins [H [7] . Particular instances of Question 2 have been solved by Jiang [HI [16] , 
who, when r = o(n/ \ogn), has shown the much more general result that M r Mj converges entry-by- 
entry to a square Wishart distribution (for which the extremal eigenvalue asymptotics are known, see 

EI)- 

We set out to examine Question 3, via Question 2; in the process, it transpired that Questions 2 
and 3 both can be dealt with simultaneously. We have obtained thus exact answers for all r, n, as 
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well as asymptotics for both r fixed, n growing to infinity, and for r, n growing to infinity in any way, 
including when r = 0{n); then we noted that all results generalize in a way which we explain below to 
yield various asymptotics of extremal eigenvalues of special /3-Jacobi ensembles. 

The /3-Jacobi ensemble, defined below for general (3 > 0, was the key to solving the MP. 

Definition 1.1. The numbers Ai, A2, . . . , A m £ [0, 1] are [3-Jacobi distributed with parameters a,b > —1 
if their joint distribution is given by 

CB.a.b.m . ~ 
^' ' ' i—l 

with cp^a^.m being the value of the corresponding Selberg integral, i.e., 



<<j 



™ r(|(o+i))r(|(6+i))r(i + fi) 

Cf3,a,b, m - II 7 TnT J- v • (1) 

f=\ r(l + |)r(f(a + 6 + m + i)) 

When p = 1,2 (corresponding to the real, respectively complex cases), the scaled eigenvalues of 
M r Mj are /3-Jacobi distributed (sec Collins [6, 7 , Sutton [28). This result is in fact more general, and 
we give here only the form in which we will use it in Section [3] 

Proposition 1.2. [following Theorem 5.1.3, ]28$ 1 Let U n be an n x n Haar- distributed orthogonal or 
unitary matrix, and let M r be the r x r upper left corner of U n . Let X\, x%, . . . , x r be the eigenvalues of 
YY T . Then Ai := X\jfi,\% :— Xz/fi, . . . , A r := x r /(3 follow the (3-Jacobi distribution with parameters 
a = and b = n — 2r ([3 — 1 in the real case, [3 = 2 in the complex one). 

Thus, the most important observation was that if one wants to find the answer to the MP, one 
should study the extremal eigenvalues of /3-Jacobi ensembles. 



1.1 /3-Jacobi ensembles and extremal eigenvalue distributions: previous 
work 

The /3-Jacobi ensembles (for [3 — 1,2) have been widely studied as the MANOVA real and complex 
distributions (see for example Muirhead [35] for the real case), and general (3- analogues have been 
studied in the context of Selberg integrals [27], [3], [21], [20] and log-gas theory (for a comprehensive 
study of the latter, see Forrester |14j). They have been given increasingly simple matrix models by 
Lippert [23], Killip and Neciu [55], and most recently by Sutton [55] (the latter are the ones used here, 
in experiments). 

Many things are known about these distributions; exact results include the cumulative distribution 
functions (CDFs) for extremal eigenvalues [TU] for all /3-Jacobi ensembles, as well as extremal eigenvalue 
asymptotics for special classes of /3-Jacobi ensembles (see Jiang [E]). In the latter paper, the author 
shows that the asymptotics for the largest eigenvalues are the same as for (special) /3-Laguerre and 
/3-Hermite ensembles, as heuristically shown by Edelman and Sutton |12j and then rigorously proved by 
Ramirez, Rider, and Virag [5S] (such asymptotics are now known as the "/3-Tracy-Widom laws", and 
they are "soft-edge"). Finally, for /3 = 1, 2, Johnstone [TH] has shown that, for a very large class of real 
or complex Jacobi ensembles, the extremal eigenvalues (at the "hard edge") have regular Tracy- Widom 
[29] asymptotics. For calculations of correlations functions (in general) and both kinds of edge-behavior 
(for [3 £ 2N) and other useful information, see Forrester |14)b 

In this paper, we examine the hard-edge (a/r — > 0) behavior of certain Jacobi ensembles not consid- 
ered before, and we find that (perhaps unsurprisingly) it agrees with the hard-edge behavior of certain 
corresponding Laguerre ensembles, some of whose limiting behavior was previously known, at least for 
certain values of /3. The difference between the hard edge and the soft edge regime in this context is 
easy to explain; it essentially boils down to {a/r, b/r) converging to positive constants (the latter), and 
one of them converging to (the former). 
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For example, the Tracy- Widom asymptotics proved by Johnstone hold for real and complex Jacobi 
ensembles ((3 — 1,2) in the soft edge regime, whose extremal eigenvalues converge to numbers in (0, 1). 
In the cases of interest for us, as we will see, a is actually constant, and thus a/r — > (hard edge) and the 
smallest eigenvalues converge to 0; unsurprisingly, the Tracy- Widom asymptotics are not valid here. In 
fact, we see the same phenomenon as for the related Laguerre ensembles: those ensembles for whom the 
extremal eigenvalues converge away from (soft edge) exhibit Tracy- Widom fluctuation asymptotics, 
but those for which the smallest eigenvalue converges to (hard edge) have either exponential ones 
(see, e.g., Edelman [TT], (3 = 1, 2), or given by a Bessel kernel (see Kuijlaars and Vanlessen [T], for work 
on modified Jacobi ensembles when (3 = 2). 

Remark 1.3. It is worth mentioning that there is a well-known simple procedure for "turning" (a,b) 
(3 -Jacobi ensembles with eigenvalues Ai, . . . , A„ into (3 -Laguerre ones with parameter a (the latter are 
(3 generalizations of Wishart), by letting the parameter b — > oo, and scaling the ensemble by letting 
A, — > Xi/b for all i. See Forrester \14\ 4- 7-1]; the limiting procedure described there for the normalization 
constant also applies to the distributions themselves. 

As a corrolary, this means that our asymptotical results also give both the exact and the asymptotical 
smallest eigenvalue distributions for the corresponding hard edge distributions in (3-Laguerre ensembles, 
whose limits have been known to exist in terms of a stochastic Bessel operator (conjectured by Edelman 
and Sutton in UStf and proved by Ramirez and Rider in J19f). Many of these limits (for various /3 and 
a) have been known; the cases we deal with here seem to be more general than anything else we were 
able to find in the literature (although for (3 = 1 or an even integer, these cases are known 114V - 

Although intuitively it makes sense that the (5-Jacobi hard edge would always correspond to the (3- 
Laguerre hard edge, one cannot move in the opposite direction, and claim that knowing the asymptotics 
for the latter will yield those for the former, since in that case the limiting procedure would have to be 
ordered: first b —¥ oo, them m — > oo. As such, our results, for which m and b + m tend to oo at any 
(dependent or independent) rates, are more general. 

The method employed here (use of generalized hypergeometric series) is the same method that we 
employed in jTUj; however, this time we apply it directly to the probability density functions (PDFs) 
rather than the cumulative distribution functions (CDFs), and as a result the hypergeometric expres- 
sions we obtain are simpler, and allow us to do an asymptotical analysis, while the expressions obtained 
in [TU] do not yield easily to asymptotic study. 

The rest of the paper is organized as follows. In Section[2] we define hypergeometric functions, Jack 
polynomials, and associated Pochhammer symbols. In Section [3j we give a general hypergeometric 
expression for extremal eigenvalues distributions of Jacobi ensembles, and in Subsections |3.3| and |3.4| we 
calculate asymptotics in some special cases which correspond to generalizations of (3 = 1 and (3 = 2. In 
particular, this is where we answer the questions asked in the first paragraph. Finally, in Section [4] we 



illustrate numerically how the distributions of Subsections 3.3 3.4 agree with Monte Carlo experiments 



2 Hypergeometric functions 

Hypergeometric functions of multiple variables, also known as generalized hypergeometric functions, 
can be defined as formal power series, just like their classical counterparts. In the multiple variable case, 
the role of the monomials is taken by Jack polynomials indexed by partitions; if X := (x\, . . . , x m ), 
the Jack polynomial C(((X) defined for a partition k and the variables X is a symmetric, homogeneous 
polynomial. These polynomials satisfy various recurrences and systems of equations. For more infor- 
mation, see Chapter 12 of [14] . which is entirely dedicated to them. For computations of such quantities 
and connections to /3-ensemble statistics, see the MOPs software package and associated paper [3], as 
well as Koev's hypergeometric function package and associated paper [23] ■ In the cases (3 = 1,2, these 
polynomials are versions of the zonal polynomials (see [25j ) and respectively scaled versions of the 
well-known Schur polynomials. 
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To define Jack polynomials, we must first introduce two new quantities: the generalized Pochhammer 
symbol (or generalized rising factorial), and the symbol 

Definition 2.1. For any k = (ki, . . . , k m ) a partition of k and for any j3 > 0, 



to^fi^-fC- 1 )) 



where (x)^ is the classical rising factorial, (x)k t — T(x + ki)/T(x). In addition, given the diagram of 
the partition k (see Figure^^, we define for every square s the "arm-length" a K (s) as the number of 
squares to the right of s, and the "leg-length" of s as the number of squares below s. Then 



/3 _ 



n (*«(*)+ +*«(*)) (us) +1+^(3 



k=(3, 1) 
a^s) = 1 
l K (s) = 

= (2/p)(l)(l + 6/p)(2 + 4/p)(4/p)(l + 2/p)(2/p)(l) 



Figure 1: Arm-length and leg- length for the square s, along with j£ 



1]' 



The Jack polynomials have various normalizations; the one that we use here has the following two 
properties: for any integer k and for k h k a partition of k, and denoting by I m the vector of m Is, 



K,\-k 



(*). 



, 2fe 



(2) 



Finally, we can now give the definition of the hypegeometric function of multiple variables, following 

El- 

Definition 2.2. The hypergeometric series p F q is given by 



p if K . . . , a p ; bl , . . . , b q ; X) ^ £) £ i ^* ' ' ' . 

For the case p < q + 1, the series converges everywhere in the hypercube {|a^| < 1, 1 < i < n}. 
We will mostly focus on the cases p = 2, q = 1 (sometimes with m = 1, in which case we recover the 
classical 2F1 hypergeometric function of one variable), and sometimes p = q= lorp = 0, q = 1- It is 
worth mentioning two more facts about hypergeometric series, the first one which we quote from |14) 
(formula 13.5 with p = 2, q = 1), and the second one being an immediate consequence of the first one 
(given the homogeneity of Jack polynomials): 



lim 2-F 1 ] b; c; X/a) 

a— >oo 

lim 2 F((a,b;c;X/(ab)) 

a.b—^oo 



lF?(b;c;X) , and 
Ff(c;X) . 



(3) 
(4) 
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3 Smallest eigenvalue distributions 

3.1 Most general case 

We are interested in finding an expression for the probability density function for the smallest (respec- 
tively, largest) eigenvalues of the Jacobi ensemble of parameters a and b (see Definition [OJ . 

We would like to obtain the marginal distribution f m i n for the smallest eigenvalue; assume A = A m . 
To that extent, we will integrate out all other variables but A TO , to obtain 

fmin(X) = -JH—XZW-HI-*)^- 1 X 
r rn-l 

/ f[Af 0+1) - 1 (l-A i )^ 6+1 )- 1 (A i -A)' 3 • TT |Aj - Xjf dXi... d\ m -i . 

Note that the m in the first numerator of the right hand side comes from choosing A m as the smallest 
eigenvalue. 

We can now make the variable change xi = jErr > which maps Ai from [A, 1] to [0, 1]. The distribution 
thus becomes 

f min (X) = A 4c+i)-i ( i_ A) #".("+".)-i x 

„ m — 1 „ 

/ TT x?^- 1 (l-x t f (l-Xiil-X))*^- 1 Yllxi-xj]" dx 1 ...dx m -i . (5) 

We will now give a different expression for the integral on the right hand side by using equation 
(13.12) from [TJ]. For our particular case, this result specializes to the following Proposition: 

Proposition 3.1. We have 



f m-l 

/ TT *i )_1 (l-^y 3 (l-a^l-A))^ 1 )- 1 Hlxi-Xjf dx 1 ...dx m _ 1 



1 f TT .3.M-1, 

Cp,b,l+2/P,m-l i[0,l]("»-i) i=1 , ; 



^(1 _ £( a + 1)^(6 + m - 1); §(6 + 2m - 1) + 1; (1 - A)7 m _i) 



of: 

21 v ~ 2 V " ' ~ n 2 K " ' "" 2' 

where Cg^ ,1+2/^, m-i * s ffwen &y ([I]) and I m -\ has the same meaning as in 

Thus, we obtain the following theorem: 

Theorem 3.2. The probability distribution of the smallest eigenvalue A of the (5-Jacobi ensembles of 
parameters a and b and size m is given by 

fmin(X) = C , aAm xiW- 1 (l-A)*"*^)- 1 X 

p2 /0 h g(o + l) P(b + m-l)J(b + 2m-l) 
2*1 \ L g ' 2 ' 2 r L,{1- A)I m -i) , (b) 

with 

m c f3, &,l+2//3,m-i 



Making the observation that 1 — A m i„ is the largest eigenvalue X ma x of the Jacobi ensemble of 
parameters 6, a, and size m, we immediately obtain the following corollary. 
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Corollary 3.3. The probability distribution of the largest eigenvalue A of the /3-Jacobi ensembles of 
parameters a and b and size m is given by 

WA) = Cp, a , b , m (i-Aj^WHA^Hx 

a/0 + <8(a + m - 1) g(a + 2m-l) r 
2^1 I 1 o . o ' o hl,Ai m _i), (7) 



with 



,m— 1 

Cp.a.b.m 



A quick examination of the distribution ^ suggests right away that, if a is fixed or grows to oo 
much slower than m or b + m, the smallest eigenvalue should go to 0. In order to obtain asymptotics 
for the hypergeometric expression, we would then like to employ a transformation that would change 
(1 — A) to A; however, for multivariate hypergeometrics, such formulae are only available when the series 
terminates (i.e., 1 — j3(a + l)/2 is a non-positive integer). In the absence of such a transformation for 
the general case (which is known in the classical case as a Kummer transformation), we can also deal 
with the case when the multivariate hypergeometric can be reduced to a classical (single variable) one. 
This can be done if the the only partitions with corresponding non-zero coefficient in the series have 
only one part, i.e., j3{a + l)/2 - 1 = -£/2. 

In the following sections, we will study the formula of Theorem |3.2| for the particular values of a, b 
and j3 that place us in one of these two cases, and find asymptotics in various regimes. 

3.2 Asymptotics for special cases 

Here we return to the motivating problem; what values of j3, a, and b are of interest in practice, and is 
it possible to use the hypergeometrics expressions given in Theorem |3.2| to find the asymptotics? 



For the MP, as given by Proposition 1.2 the distributions correspond to: 

Case 1. The real case: ft = 1, a = 0, b = n — 2r, and m = r. What turns out to be crucial in this case is 
that /3/2(a + 1) — 1 = —,8/2; we will therefore study the general case a = 2//3 — 2, m = r. 

Case 2. The complex case: (3 — 2, a = 0, b = n — 2r, and m = r. For this case, the crucial factor is that 
/3/2(a + 1) — 1 = 0; this is a particular instance of (3/2(a + 1) G N + , the positive integers (or 
natural numbers). We will analyze this latter, more general case. 

Remark 3.4. It is worth noting that, although not connected directly with the MP, another case of 
interest relates to the quaternionic Haar measure. As per if instead of an orthogonal or unitary 
matrix in the MP we start from a symplectic ( quaternionic ) one, the resulting distribution is the same 
as in \l.S\ with /3 = 4, a = 0, b = n~ 2r, and m — r. This can be seen as a particular instance of Case 
2; so Case 2 will cover both complex and quaternionic matrices. 

In the following sections we study the asymptotics of the smallest eigenvalue distributions in each 
of these two cases. 

3.3 Case 1: a = | - 2 

In this case we have that /3/2(a + 1) — 1 = — /3/2. We start by examining the smallest eigenvalue 
distribution as given by Theorem |3.2| 



fminW = Cp, b , m A"f (l-A)f m ( 6+m )- 



F 2/p ( (3 P(b + m-l)J(b + 2m-l) 

2*1 kg" I 2 ' 2 ~ A ) 1 m-l) , (O) 
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here 



c @,2//3-2A,m 

After various cancellations, we obtain that 



m(b + m) r (§ (b + m - 1) + l) r (§ (m - 1) + l\ 



r(i-|)r(f(6 + 

Lemma 3.5. We have 



2m - 1) + 1 



pf) fP f g(b + m-l) ; /3(b + 2m-l) + 1; (1 _ A) m-A = ^ / ^(ft + m-l^ g(m-l) . /3(b + 2m-l) + ^ _ A 
V 2 2 2 / V 2 2 2 / 

where the second hypergeometric is no longer the matrix argument one, but the classical one (corre- 
sponding to matrix dimension 1 ). 

Proof. The proof is based on two simple facts. One is that the series is absolutely convergent for 
A G [0, 1], and the second is that for any partition k of k, 



where the Pochhammer symbol in the right hand side is the classical falling factorial. As soon as k has 
more than one part, the contribution from that part will make the falling factorial by Definition |2.1| 
We can now see that 



/3\ / 0(b+m-l) 



g(P f3(b + m-l) l3(b + 2m-l) A x ^ \*)k V 2 J k p 

1 U' 2 ' 2 + 1 '( 1 - A ) ) = 2-, fe , ^Q+am-D T^" C W ((1 ~ A)Jm - x) 



(A (i 



' )(b+m-l) 



Using ([2]) for ft = [k] , along with the fact that in this case j[k] can be easily computed to be 
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2, , nl OS2k M ^ 



[k] -(2/t3) k k!Y[(l + -( t -l)) =(2/pf k k\ ^ 

after cancellation, Lemma |3.5| is proved. □ 

To analyze the classical hypergeometric on the right hand side of |3.5[ we make use of the following 
transformation which can be found for example as formula 15.3.6 in [2]: 

2 F 1 (a,b;c;z) = ^°^ C ~ " ~ b J 2-F1 (o, b; a + b - c + 1; 1 - z) + 
1 (c — a)l (c — oj 

(1 _ z)c -a-6 r(c)r(a + b - c ) zFi(c _ c _ b . c _ fl _ 6 + 1; ! _ ^ _ (Q) 

r(o)r(6) 

By plugging in a — P( b+ ™'~ 1 ) ; j, — ff(m-i) ^ an( j c _ g(fc+2m-i) _j_ ^ ag as ^ = 1 _ we obtain 

A 1+ ^ B W , m 2 ^ + 1, + 1; 2 + § A) do) 



7 



with 

Ap,b, m = — s t4t. — - =r- and 



/?(H-2m-l) +1 1 F /| + i 



r ( ^™ + 1 1 r ' ff( b + m ) 



2 ' / V 2 



r ( ^2=11 + 1) r (-1 - 1 



B,9 ^' m F f fl(m-l)\ r ^(b+m-1) 
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The useful thing about the right-hand-side formula is that m and b + m act like essentially independent 
variables. If we combine ([8]), the results of Lemma |3~5j and ( flO| , followed by factoring out 

F := 



r( /?(i,+2m-i) +1)r(1 + | )r( _|) 



r(^±^ + i) 



from the right hand side of (10), we can conclude the following. 

Theorem 3.6. The smallest eigenvalue of (3-Jacobi ensembles with a = -| — 2 is given by 

//3,6,m(A) = C Pjb , m \- 0/2 (1 - A)"™^/ 2 - 1 

1 ff(6 + ro-l) g(m-l) /? 



r(-f)r(^p + i) v 2 2 ' 2 

A 



.1+/3/2 1 1 \ 2 hi 



^ 2 ^ + b^ + l;2 + f;A)) (11) 



whe 



r(-f)r(i + f) * r (f(6 + m-i) + i) , 

C P , b , m = V 7 \v ' gm(b + m) -A^ (f(m-l) + l 

We can now analyze the asymptotics in various regimes. 

Regime 1: m fixed, b — > oo. This is the case when the /3-Jacobi ensembles "turn" into /3-Laguerre, and so 
we are also computing here the exact distributions for the smallest eigenvalue distributions for 
/3-Laguerre ensembles. 

For this case, we use the following two facts: 

r(a; + a) 

— - — — — > 1 as x — s- oo and a is fixed (12) 

x a T(x) 

X 

2-Fi(a, b; c; — ) — > iFi(6;c;z) as a — > oo . (13) 
a 

If we make the transformati on y = /3(b + m) A/2 and allow 6 — > oo, the hypergeometric expression 
between the parentheses in (fill) becomes 



Fi(fl^;-f;y) , )1+ s,2 + 1;2+ f ;y) -sin(^) / g(m-l) _P 

<(-§)r(fr* + i) w r(f + 2)r(«j - V 2 ' 2' y 



where we used formula 13.1.3 from [2] 
We obtain thus the following result: 
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Theorem 3.7. As b — > oo while m is kept fixed, the limiting distribution for the scaled smallest 
eigenvalue (f3(b + m)\/2) of the fi-Jacobi ensemble with a = 2/ f3 — 2 is given by 

Note that when (3 = 1, this agrees with [Jjj] and [TT] (also taking into account that our scaling 
differs by a factor of 2 from the ones used there.) 

Regime 2: m, (b + m) —} oo. Note that this also covers the case b fixed. We now use facts (12) and (13), as 
well as 

2 F 1 (a,b;c;^-)^ F 1 (c;x) (14) 
ab 

to obtain the limiting distribution of y = /3m(b + m)A/2. 

We shall use the fact that the hypergeometric expression in ( 11 ), when multiplied by r(/3m/2 + 1), 
in this regime, converges to 

i „ ( p p \ ( p Y 13 ' 2 i _ .„ p p 



G ? {y) ■= pT^lJ ° Fl {- 2 ; 2 V ) \2 V ) r^lj ° Fl{2 + 2 ; 2 V) ; 

using the definition of the classical o-Fi hypergeometric function as well as formulae 9.6.2 and 
9.6.10 from [2], we obtain that 



sin f f 



2sinf#7r) ,p xl+| 



= I ~V ) K 1+§ (Y2py) , 

where K v is the well-known modified Bessel function. 
Using this we can now conclude the following. 

Theorem 3.8. As m, (b + m) — > oo, the limiting distribution for the scaled smallest eigenvalue 
(Pm(b + m)\/2) of the P-Jacobi ensemble with a = 2/P — 2 is given by 



M = -^p- (£P r(i + tWe-*K 1+i {VWv) ■ 

Note that when P = 1, this once again agrees with the combined results of [15] and [TT]. Our 
scaling differs from the one used there by a factor of 2; in effect the distribution for this case is 
simply 

fM = ^e-y-^y . 



The importance of this result is that the limit is achieved regardless of how (6 + m) and m grow 
to infinity. In the case of the MP, we have m = r, b = n — 2r, and the condition is then that both 
r and n — r grow to infinity. This is stronger than the results following |15j , where the condition 
is that r = o(n/ log n). 



3.4 Case 2: (a + 1)0/2 G N+ 

Note that this case covers the P £ 2N + case analyzed in [TT] (although it is true that Forrester computes 
not only the hard edge/soft edge limits for P even, but also all limiting correlations). 

For this particular case we will use a different hypergeometric formula to express / m (A), which will 
yield a terminating series, which can then be converted into an analyzable expression. 

Let (a + l)P/2 = k, with k a positive integer. 

Instead of (13.12) from [TT], this time we use equation (13.7) from the same, which in this particular 
case specializes to the proposition below. 
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Proposition 3.9. We have 



m — 1 



1 T x^ 1 ^ 2 - 1 (1 - Xl f (1 - Xi (l - A))*" 1 TT 1^ - xtf rfxi . . . cte m _i = 

C/3,6,l+2//3,m-l J [0,l]< m - jL"l i<j 

2^(1 -m,-m- 6+1; -2m -&+ 1 - ^{l-A}*" 1 ) , 

where cp t 2k/p-i,b,m-l is 9 iven by Q- 

Note that the hypergeometric series is terminating; in fact, it is a polynomial of degree at most 
(to- l)(jfc- 1). 

Also note that since b > — 1, m > 1, and j3 > 0, 2to + 6— 1 + % > m — 1, the series is well-defined 
(if, for some k, (—2m — 6 + 1 — = 0, then necessarily (1 — m) K = as well, and so the potentially 
"offending" term in the series does not in fact exist). 

Under these conditions, using Proposition 13.1.7 from [TS] . we can change the variables in the 
hypergeometric expression from (1 — A) 1 to A 

p4 /, n . x1 _ ,^ 2., ufc _. ^ (1 - m, -m - 6 + 1; 2 + §(fc - 1); {A} fc-1 ) 

2-F1 (1 — m, — m — 6 + 1; — 2m — b + 1 — -HI — A^ = — ttb 

11 P X ' ' 2 F 1 4 ^(l-m,-m-6+l;2+|(fc-l);{l}^i) 

According to equation (13.14) of the same [T4] . 

? fe_1 r (2 + §/) r (2m + b+ §/) 

2 F^^(1 - to, -to - 6+ 1; 2 + -(fc - 1); {l}^ 1 ) = TT LJ y A ? == A Mk . 

P J= i r(m + l + fj)r(m + 6+l+ |ij 

We now group the left and, respectively, right terms in the fraction above, and rewrite the product 
as follows: 

k— 1 m— 1 _^ ^ 

A m ,b,f>,k = n n ( m + b +*+ -sj) 
j=\ i=i 1 + 1 + p j p 

k— 1 m— 1 1 p 

S5M^ (( " +,+,, i +fl 

r(l + (i + l)f) r(fc + §(m + 6 + i)) 



m— 1 

n 



r (*! + (* + 1)1) r(i + f(m + 6 + t)) 

Putting everything together, we obtain 

Theorem 3.10. For the case when (3/2{a + 1) = k € N+, i/ie smallest eigenvalue of the (3-Jacobi 
ensemble is given by 

fm(X) = W m , b ,f i , k \ k - 1 (l-\)$ m <!* m )- 1 

x 2 F 1 4//3 (l-TO,-TO-6+l;2+|(fc-l);{A} fc - 1 ) , (15) 

with 

m C0,b,l+2/p,m-l 



c P,2k/l3-l,b,m,A m ,b,p,k 

After appropriate cancellations, we obtain that 



r(i+f] mri 



'(Wfm) T(k + Ub + m)) 

W M k = V ( ■ f - / • -^jz ^ • (16) 

r(fc)r(fc+f) r(i + fm) rf§(6 + m)J 
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As in the previous case, we can now start to analyze the asymptotics in two regimes. 

Regime 1: to fixed, b — > oo. Using ([3| and (12), we can obtain the distribution of the scaled eigenvalue 
y = (b + to)A, as follows. 

Theorem 3.11. As b — > oo while m is kept fixed, the limiting distribution for the scaled smallest 
eigenvalue ({b + m)\) of the (3-Jacobi ensemble with (3(a + l)/2 = k G N+ is given by 

^ r(i + | N ) mrffc + fm 



Uv) - v J > -V • r r v y k " e ~" mv/2 rf'Q- - m; 2 + > - 1) ; {- y }^) . 
r(fc)r(fc + §j rn + fmj p 

Note that in the case (3 = 2, fc = 1, which is the case 2 of the MP, the hypergeometric above 
simply becomes a constant which cancels the more complicated terms of the normalization, and 
yields that 

f m {v)=me- m y/ 2 , 

which is consistent with the results of [15] and [11] . 

Regime 2: to, (b + to) — > oo. This covers the case when b is actually fixed. Again, using (3), @, and £2), 
the distribution of the scaled eigenvalue y = m(b + m)X is given below. 

Theorem 3.12. As to, (6 + m) — > oo ; £/ie limiting distribution for the scaled smallest eigenvalue 
(m(b + m)X) of the (3-Jacobi ensemble with P(a + l)/2 = k G N+ is given by 

M = W / J } ■ y k - 1 e-^\F^(2+ 2 (k~l);{y}^). 



Once again, when (3 — 2 and k — 1, the hypergeometric expression in the above is constant, and 
the answer is 

f(v) = e~ v . 

Again, this is consistent with [T5] and [TT]. The advantage over [T5] is that when b = n — 2r and 
to = r, as is the case in the MP, there are no conditions over the growth to oo of r and n — r; in 
|15j . r must be o(n/ log n). 



4 Numerical Experiments 

We have tested numerically all the formulae obtained in the previous sections. All plots were obtained 
in MATLAB; to compute the hypergeometric functions involved we have used Koev's easy-to-use hy- 
pergeometric series package [531 E|- F° r the Monte Carlo tests, we have used the convenient to work 
with bidiagonal models for /3-Jacobi distributions, given below: 



J, 



B 



B 



P,n,a,b ' Bp n a b , 
C-n S n C n _i 
Cn-l s n _l 



where 



-S2Cj 

cis[ 



where all Cj's and c^'s are independent variables distributed as follows: 



iBth, I |(o + i), |(& + ») 



; /7 ' I ^(a + 6+ 1 + i) 
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and Si = y/\ — cf , for all 1 < i < n, while s\ = — c' 2 , for al 1 < i < (n — 1). Here Beta(s, t) stands 
for the well-known distribution with pdf proportional to x s_1 (l — a;)' -1 on [0, 1]. 

This matrix model, together with a proof that the matrix Jj3^ n . a ,b has eigenvalue pdf given by the 
Jacobi ensemble with given parameters, is a beautiful result of Sutton's |28j . 

Figure [2] is an illustration of Theorem 3^ The solid red line corresponds to the exact distribution 
of the smallest eigenvalue, as given by the theorem, for the case when ft — 1.75, m = 4, a = 2.3, and 
b = 2.5. The normalized histogram (obtained using the histnorm) represents the results of 10, 000 
Monte Carlo tests using the bidiagonal matrix models. 



Smallest eigenvalue distribution at m=4, p = 1 .75, a = 2.3, b = 2. 




0.1 0.2 0.3 0.4 



Figure 2: The solid red line represents the theoretical distribution, while the normalized histogram represents the 
results of a Monte Carlo experiment with 10,000 trials, using Jp. n , a ,b- 

Figures |3j and |3j represent a comparison between the theoretical results of Case 1: a = j| — 2, 
respectively, Theorems |3.7| and |3.8| (illustrated by the solid red lines), and Monte Carlo experiments with 
10,000, respectively 5,000 trials (represented by the normalized histograms). Note that convergence 
occurs fairly quickly; for Figure |3j which considers the case b — > oo, the histogram of (3(b + m)/2X min 
is very close to the plot of the asymptotical distribution, even though b = 10. For Figure |4j we only 
need to take 6 = 5 and m = 5 to see how close the histogram of (3m(b + m) /2A m i„ is to the solid line 
representing the asymptotical distribution. For this latter case, also note the singularity at 0; this is 



caused by the singularity of the Bessel function in the asymptotical formula, as given by Theorem 3.8 



Monte Carlo vs asymptotical distributions for |5(b+m)/2 >. 
with m = 4, b = 10 (resp., <*>), (3 = 1/3, and a = 2/p -2 




0.2 0.4 0.6 0.8 1 



Figure 3: The solid red line represents the asymptotical (b — oo) distribution, while the normalized histogram 
represents the results of a Monte Carlo experiment for b = 10, with 10, 000 trials. 
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Monte Carlo vs asymptotical distributions for |5m(b+m)/2 k 
with m = 5 (resp., ~>)> b = 5 (resp., °°), p = 1 , and a = 2/p - 2 




0.2 0.4 0.6 0.8 1 1.2 



Figure 4: The solid red line represents the asymptotical (m, b = oo) distribution, while the normalized histogram 
represents the results of a Monte Carlo experiment for m = 5, b = 5, with 5, 000 trials. 



Finally, figures |5j a nd ^r epresent comparisons between the theoretical results of Case 2: a = ^ — 1, 
specifically, Theorems |3.Tl| and |3.12| The exact distributions given by the theorems are represented by 
the solid red lines, while the histograms are the results of Monte Carlo tests with 10, 000 trials. This 
time, one has to increase j3 to 50 to obtain significant results; however, since the distributions represent 
asymptotics, that is still rather small. Figure [5] covers the case (3 — > oo, that is, Theorem |3.11| and the 
histogram is of the distribution on (b + rn)A m i„, while Figure [6] covers the case m, j3 —¥ oo (Theorem 



3.12 1, and the histogram reflects the empirical distribution of m(b + m)A r< 



Monte Carlo vs asymptotical distributions for (b+m) k 
with m=6, b = 50 (resp., ~>), p = 1.5, k = 2, and a = 2k/p - 1 




0.5 1 1.5 2 2.5 3 



Figure 5: The solid red line represents the asymptotical (b = oo) distribution, while the normalized 
histogram represents the results of a Monte Carlo experiment for b = 50, with 10, 000 trials. 
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Monte Carlo vs asymptotical distributions for m(b+m) X . 
with m = 15 (resp., ~)> b = 50 (resp., ~), p = 3.33, k = 2, and a = 2k/p - 1 




Figure 6: The solid red line represents the asymptotical (m, b = oo) distribution, while the normalized 
histogram represents the results of a Monte Carlo experiment for m = 15, b = 50, with 10, 000 trials. 
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